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A unified theory is presented for data- aided equalization of digital 
data signals passed through noisy linear dispersive channels. The 
theory assumes that some past and/or future transmitted data sym- 
bols are perfectly detected. We use this hypothesis to derive the 
minimum mean-square error receiver. The optimum structure consists 
of a matched filter in cascade with a transversal filter combined with 
a linear intersymbol interference canceler which uses the ideally 
detected data symbols. The main result is an expression for the 
optimized mean-square error as a function of the number and loca- 
tion of the canceler coefficients, the s/n, and the channel transfer 
function. When the number of canceler coefficients is zero, we get the 
well-known result for linear equalization. When the causal or post- 
cursor canceler approaches infinite length, we obtain the well-known 
decision feedback result. When both the precursor and postcursor 
cancelers become infinite, we obtain the very best result possible, 
namely, the matched-filter bound dictated from fundamental theoret- 
ical considerations. Neither the decision feedback nor the matched- 
filter results can be achieved in practice since their implementation 
requires infinite memory and storage. Our theory can be used to 
calculate the rate of approach to these ideals with finite cancelers. 



I. INTRODUCTION 

The theory of linear and decision feedback equalization to mitigate 
the effects of intersymbol interference (isi) and noise in digital data 
transmission is well known. 1-4 In this paper, the problem of equalization 
is cast in a general framework of an isi canceler aided by past and/or 
future data values. This general structure is suggested from optimal 
detection theory and is shown in Fig. 1. The optimal detector of digital 
data in the presence of additive Gaussian noise and isi is comprised of 
a matched filter and an isi estimator which is used to cancel the 
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Fig. 1 — Block diagram of data-aided equalization. 

interference. 5,6 The implementation of this structure is often impract- 
ical because of its complexity. 7,8 

In our theory we postulate that some portion of the isi can be 
perfectly synthesized and, therefore, subtracted from the incoming 
signal. In other words, we replace the optimal estimator with a practical 
one. The effect of the remaining interference is then minimized by a 
linear filter or a conventional linear equalizer. In practical systems, 
however, perfect estimation cannot be achieved; therefore, our results 
serve as ideal limits. The inclusion of occasional errors in our theory 
has proved mathematically intractable so far. 

In Section II, we determine the minimal mean-square error (mse) 
when an arbitrary set of data symbols is known to the receiver. In 
Section III, the optimal receiving filter is derived and analyzed. The 
performance of the infinite linear equalizer, the decision feedback 
equalizer and the infinite canceler are obtained as special cases of the 
general result. Section IV covers a discussion of numerical results. 

II. MINIMUM MSE FOR DATA-AIDED EQUALIZATION 

In Fig. 1, the transmitter generates the data sequence {a„} whose 
elements are assumed to be independent identically distributed (i.i.d.) 
discrete random variables. These discrete amplitudes sequentially 
modulate the pulse p(t) at a rate 1/T to produce the transmitted 
signal. The pulse shape, p(t), can be viewed as the overall impulse 
response of the transmitting filter and the transmission channel. White 
noise, v{t), is added to the received signal which is then applied to the 
linear receiving filter, w(t ). The output signal is sampled at the symbol 
rate 1/T and combined with the output of the canceler. The linear 
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canceler is modeled as a transversal filter with coefficients, {c„} , where 
n E S, and where the set of integers S denotes the range of the 
canceler's taps. 

For most applications, and for the special cases investigated later in 
this Section, the range will contain the neighboring taps of the refer- 
ence location but never the reference location itself, and consequently, 
S = {— Aft, • • ■ , — 1, 1, • • • , iV 2 }. The canceler operates on the past 
received symbols a n -i, • • • , a n -N 2 and on the future received symbols 
a„+i, • • • , dn+Nt, which are assumed to be known to the receiver. 
Clearly, to realize an operation on the future data symbols, a time 
delay of at least NiT seconds has to be introduced. 

For a general set S the output signal x n = x (nT) can, thus, be written 
as 

00 

Xn = £ r k a n -k - £ Ckdn-k + £n . (1) 

k— oo Aes 



Where r« = r(kT) is the overall impulse response evaluated at t = kT, 
r„=T j w(r)p(kT-T)dr, (2) 

and where £* = £(kT), i.e. 
&- [ w(r)v(kT - r)dr, for k « -oo, . .. , -1, 0, 1, . . . , oo. (3) 



-I 



To facilitate modeling of various types of linear modulation schemes, 
the data sequence, the noise, and all impulse responses are assumed to 
be complex valued. In general, p{t) will be the preenvelope of the 
passband transmission system with respect to a carrier frequency. This 
notation has become extremely useful and economical in this field. 9 
Specifically, it permits a unified presentation of baseband and pass- 
band systems. 

The output signal, x n , after slicing or quantizing is usually taken to 
be an estimate of the transmitted data symbol a n . Our goal now is to 
obtain a receiving filter, w{t), and canceler taps, {c„}, so that the mse, 

e = E{\xn-a n \ 2 }, (4) 

is a minimum. To determine the optimal canceller coefficients, {c„}, 
we differentiate eq. (4) with respect to c, n G S, and set the result to 
zero 

— = al[c* - r * + c„ - r„] = 0, for n G S , (5) 

ac n 

where 
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E{a n at) = o 2 Jn*, (6) 

and S n ,k is the Kronecker delta. The immediate conclusion from eq. (5) 
is that for n E S 

c n = r n . (7) 

Inserting this into eq. (1), we get 



x n = £ r k a n - k + w(r)v(nT - r)dT . (8) 

Thus far, our approach is perfectly obvious. By knowing the data 
symbols for all integers k G S, it is possible to synthesize the resulting 
isi associated with these symbols and subtract it from the current 
signal sample x n . If the set S contains all integers k < n, we use all the 
already-decided-upon data symbols (available at the receiver without 
delay) to synthesize the postcursor isi. This is precisely what is done 
in decision feedback equalization. If the set S contains all the integers, 
except the one associated with the present instant, n, all isi is elimi- 
nated. But this, of course, requires infinite delay. In practice, the set S 
will be finite and our main concern will be to determine how it 
influences the mse. 

We now proceed to optimize the receiving filter, w(t), for a given set 
S. Inserting eq. (8) into eq. (4) and using eq. (6), the resulting mse can 
be expressed as 



e = a 2 a 



I \r k \ 2 -r -rS + l + o 2 \, (9) 

MS 



']■ 



where 



and where 



E{r{t)p(t + r)*}-o*T8(r), (10) 



a 2 



a 2 = T-£ \w(t)\ 2 dt. (11) 

CTa J-a> 

We remark that more general noise covariances can be included, but 
the calculations become more cumbersome without yielding additional 
insights. 
To obtain the optimum w(t), let 

w{t) = m(t) +\ji(t) (12) 

and define 



-'£ 



Uk=T\ W (T) P (kT-T)dT, (13) 
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where Wo(t) is the optimum impulse response of the receiving filter 
and where the Uk are the samples of the optimized overall impulse 
response. It follows immediately from eq. (2) that 



I 



r k = Um + XT \ n(T)p(kT - T)dr . 



When 



de 



= 



(14) 



(15) 



x-o 



is calculated from eq. (9), we obtain an equation for the optimum 
receiving filter: 



where 



w (t)N = p(-t)*- £ U k p(kT-t)*, 

kpS 



No— j. 



(16) 



(17) 



The interpretation of eq. (16) is standard: the optimum receiving filter 
is comprised of a matched filter p(—t)* in cascade with a transversal 
filter having taps only at those locations where the canceler has none. 
This structure is shown in Fig. 2. 

To obtain our central result, an expression for the optimal mse, we 
multiply eq. (16) by Two(t)* and integrate from — oo to +oo. This yields 
with the aid of eqs. (9), (11), and (13) the result 

€ op t = aid - Uo) • (18) 

The explicit determination of Uo is the subject of the next section. 
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Fig. 2 — Optimal receiving filter, wo(t). 
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III. THE OPTIMAL FILTER 

To determine the sample values, {U m }, of the optimal overall 
impulse response, define the autocorrelation function of the channel 
impulse response as 



-'£ 



R m = T \ p(mT-T)p(-r)*dr (19) 

and its Fourier transform as 

R(co) = i R m e- JmuT . (20) 

m=— oo 

After multiplying eq. (16) by Tp(nT — t) and integrating from — oo to 
+oo, we get the following system of linear equations in {U m }, 

U m N = R m - I U k R m -k for all m. (21) 

To determine the optimal receiving filter, we only need to know U m 
for m £ S. From eq. (21) we extract the equations necessary to 
determine Uo and partition them as follows 

U (No + Ro) - Ro - £ UkR-k for m = 0, (22) 

£ U k M m -k = (l-U )R m for m£J, (23) 

where we defined 

M k = R k + Na8k,o, (24) 

and where J is the set S augmented by m = 0. 

Note that the indices of the unknowns and the indices of the right- 
hand sides of eq. (23) have gaps of the same size and at the same 
locations. (See the definition of J above.) Thus, the set of equations in 
(23) is not in a standard form and the solution technique is not obvious. 
In Appendix A we develop a technique to solve this infinite set of 
equations with finite gaps. It involves the solution of a special infinite 
set of equations without gaps. To compensate for the gaps, we augment 
the original set of equations. Specifically, we add for m e J a finite 
number of equations to the infinite set such that the solution vanishes 
for m E J. From Appendix A, we determine that the optimum mse 
becomes 

^-" l N^Ho' (25) 
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where H is determined from the following set of equations: 

X Mkl m H m = 8 k ,o-N Mk, for k E J (26) 

me J 

and where Ml 1 is the inverted sequence of Mk, i.e., it satisfies 

£ M„M*1„ = S*.o for all£. (27) 

n 

In the following section, we investigate the minimal mse for some 
special cases. As mentioned initially, we use the realistic assumption 
that the set S contains the neighboring locations {—Ni, • • • , -1, 
1, • ■ • , N 2 ). Then J = {— Ni, • • • , N 2 ) and the coefficient matrix in eq. 
(26) is a finite Toeplitz matrix. The solution of eq. (26) and, thus, Ho 
is unique and is guaranteed to exist when R (co) + N is bounded away 
from zero and infinity. 10 These conditions are very mild and are 
satisfied in most cases of practical interest. 

3. 1 Infinite length equalizer 

For Ni = N 2 = the set J includes only the zero integer and all 
canceler coefficients vanish. Consequently, eq. (26) degenerates to a 
single equation 

H Mo 1 = l-NoMo\ (28) 

Solving this for H and inserting it into eq. (25), we obtain the standard 
result for the optimum linear equalizer, 2 

where we expressed Mo 1 in terms of its Fourier transform 

3.2 Matched-filter 

When Ni = N 2 = °° the set J is infinite; i.e., the canceler subtracts 
all the isi. Equation (26) in this case yields 

Y MJiUHm + No8m.o) - 8 k ,o for all A. (31) 

m— — oo 

Comparing this with eq. (27) gives the result 

Mm = H m + No8 m ,o . (32) 

From eqs.(32) and (23) it follows that H m = R m for all m. Thus, the 

DATA-AIDED EQUALIZATION 2029 



optimum mse for this case is 



No 



6opt = a 'NoTi£- (33) 

This is recognized as the matched-filter bound for the optimal detec- 
tion of a known signal in noise and in the absence of isi. 

3.3 One-sided canceler of infinite length 

For Ni = and N2 = oo the canceler performs as an ideal decision 
feedback equalizer of infinite length. In Appendix B the mse is derived 
for the more general case Ni ¥* 0, i.e. for a decision feedback equalizer 
with a limited number of noncausal taps. The result is 

2 Wo ,oa\ 

€opt = O a "Jyj , \o4; 

2 \vt\ 2 

k-0 

where the coefficientsMjt are determined from the following equation 

Y 1 M + k Mn- k = M m for allm. (35) 

A-0 

Here, {M*} is the causal "root" of the two sided sequence {M*}. It 
satisfies Mt = for k < and Mt = (M-k)* for k > 0. It Ls shown in 
Ref . 3 that 

\Mi | 2 = No exp \L[ In (^ + l\ d J , (36) 

and when this formula is inserted into eq. (34) we get the well-known 
result for the decision feedback equalizer, 

•w/T 



Copt = Oa exp 



Unfortunately, there is no similar simple expression for |M*| 2 , for 
k j* 0; therefore, we are forced to numerically factor the two-sided 
sequence {M*} into its causal and anticausal root. 

IV. DISCUSSION OF NUMERICAL RESULTS 

In this section, the minimal mse of data-aided equalization is eval- 
uated numerically for certain channels and for various sets of canceler 
taps. We will exhibit and discuss the behavior of the mse, CoptCMi N 2 ), 
as a function of Ni and iV 2 for typical telephone channels. As a point 
of reference, note the following easily proved inequalities: 

e opt (0, 0) > €o P t(0, 00) > €001(00, 00). 
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In the following, we examine three types of cancelers: 

(i) Starting from the infinite length linear equalizer whose mse is 
€o P t(0, 0), we increase the number of causal canceler taps; i.e., we 
examine e op t(0, N 2 ) for N 2 = Q = 0, • • • , 15. 

(ii) Starting from the infinite length linear equalizer, we increase 
the number of known data symbols alternating between causal and 
noncausal ones; i.e., we examine e op t(Ni, N 2 ) for Q = Ni + N 2 = 
0, • • • , 15 and where N x - N 2 for Q even, N 2 = Ni + 1 for Q odd. 

(Hi) Starting from the irifinite length decision feedback equalizer 
whose mse is € opt (0, 00), we examine the behavior when noncausal taps 
are added; i.e., e op t(Ni, <») for Ni = Q = 0, • • • , 15. 
Equation (25) is used to determine the mse for cases (i) and (ii), where 
Ho is obtained as the solution of eq. (26). To determine the sequence 
{Af* 1 } for all k, we observe that the Fourier transform of the sequence 
{R k } is related to the overall transfer function of the channel as follows: 



*(«)- S 



k— 



Plu-k — 



(38) 



Clearly, R (10) is periodic with period 2-n/T, and it is only dependent on 
the magnitude of the overall channel transfer function, P(w). There- 
fore, phase distortion in the channel has no effect on the mse. This is 
reflected in the well-known fact that phase distortion can be perfectly 
equalized without noise enhancement. Therefore, the sequence, 
{Mk 1 }, is obtained as follows 

T r*/r e Jk»T du3 



!* = 



2ir )_ n/T R(u)+No 



(39) 



where j = >/— I. Fast Fourier transform techniques are used to evaluate 
eq. (39). Numerical tests show that it suffices to take 64-128 samples 
of R (<o) + No in the interval {—it/T, tt/T~\. The fact that the coefficient 
matrix in eq. (26) is positive definite and Toeplitz, makes it possible to 
obtain the desired solution, Ho, recursively. This is done with the 
Levinson algorithm. 11-14 

For case (Hi), we evaluate eq. (34). The sequence, {M t }, is obtained 
from the following approach. 3 First determine {F k } for all k such that 

ln(M(w))= S F#~ ihuT - ( 4 °) 

A— ex) 

Then it follows that 

M(io) + = exp J £ F*e-^ r j, (41) 
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and 

T r /r 
Mi = — M(u>) + e jkuT da . (42) 

2,r J-„/r 

Fast Fourier transforms are used to obtain consecutively {F*}, M(«), 
and Mk. The overall channel power transfer function, |P(«)| 2 , is 
assumed to consist of a raised cosine shaped transmitting filter with 
relative excess bandwidth, a = 0.15, 

|T(«)| a 



0.5 II -sin (•-£)£ 



0. 5 [l + sin(<o + ^ 







for |w|< (l-a)ir/T 

for (1 - a) ^, < oj < (1 + a) £ 



for - (1 + a) ^< to < -(1 - a) ^ 
elsewhere 



and cascaded with the channel power transfer function | G(co) | 2 . 

Figure 3 shows two different channel power transfer functions, 
| G(u) | 2 , which are used to derive the subsequent numerical results. In 
(a) we show the equivalent baseband transfer function for the worst 
channel meeting the basic conditions of private lines (BASICBAD). 15 
Part (b) shows a transfer function (CABLE) with linearly increasing 
attenuation. The parameters Pi and P2 indicate the attenuation at 
w = —it/T and u = tt/T. A model for a baseband cable channel is 
obtained when Pi = P 2 . 

Figure 4 shows the mse as a function of the number of canceler taps 
for the various channel transfer functions and for s/n of 20 dB at the 
receiver input. The dotted line represents type (i) canceler; the dashed 
line, type (w); and the solid line, type (Hi). The curves for types (i) 
and (ii) start at the minimal mse for the infinite length equalizer and 
the curve for type (Hi) starts at the minimal mse of the infinite decision 
feedback equalizer. 

As can be observed, all curves converge very rapidly to their asymp- 
totes. The curve for type (i) indicates that only 3 causal coefficients 
suffice to closely approximate the performance of an infinite decision 
feedback equalizer. The curve for type (ii) suggests that a total of 6 
coefficients (3 causal and 3 anticausal) results in a performance which 
is very close to the optimal (the matched-filter bound). The curve for 
type (Hi) reaches very close to the mse obtained from the matched- 
filter bound with only 3 noncausal coefficients, in addition to an infinite 
decision feedback equalizer. These results are virtually independent of 
the channel involved. 
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(a) 




(b) 
Fig. 3— Power transfer functions, (a) BASICBAD channel; (b) CABLE channel. 

The channel, however, influences the best mse which is obtainable 
with the infinite equalizer, i.e. € op t(0, 0), and with the infinite decision 
feedback equalizer, i.e. € opt (0, a>). Table I shows these figures for the 
various channels. 

The minimal mse obtained from the matched-filter bound is —20.04 
dB. Therefore, 1.8 to 4.7 dB can be gained for the channels considered 
if a canceler of three causal and three noncausal coefficients is included. 



APPENDIX A 

Solution of an Infinite Set of Equations with Finite Gaps 
Let 

M k = R k + No8k,o , 
and consider 

2 UkM m - k = (1 - Uo)R m for m£J, 

k4.J 



(43) 
(44) 
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-18 




-21 




-15 



D CAUSAL TAPS, /V, =0, N 2 = Q. FOR TYPE (/') CANCELER 

\ O CAUSAL AND NONCAUSAL TAPS, 

ti Q = /V, +A/ 2 . FOR TYPE (//) CANCELER 

A DECISION FEEDBACK WITH NONCAUSAL TAPS, 
N y = Q, N 2 = oo FOR TYPE (/;/) CANCELER 



-21 



(C) 




Q COEFFICIENTS 



Fig. 4 — Mean-square error for data-aided equalizations, (a) BASICBAD channel; (b) 
CABLE channel, P t = P 2 = -10 dB; (c) CABLE channel, Pi = -10 dB, P 2 = -20 dB. 
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Table I — Minimal mean-square error 

oo Decision 
Feedback 
oo Equalizer Equalizer 
Mean-Square Error € opt (0, 0) in e opt (0, oo) in 
Channel dB dB 



BASICBAD 


-18.05 


-19.16 


CABLE 






Pi = 10, P 2 = 10 


-18.22 


-19.13 


P l = 10, P 2 = 20 


-15.35 


-17.97 



where J is a finite set. It always contains the number zero but is 
otherwise arbitrary. Equation (44) is an infinite set of equations with 
a finite gap in both the indices of the unknowns U k and the right-hand 
sides R m . Notice that eq. (44) reduces to a discrete convolution and, 
therefore, is easy to solve if k and m are allowed to take on all the 
integers, or if the gap could be removed somehow. 
Now consider instead of eq. (44) the following set of equations 

£ VkM m -M=(l-Uo)(R m -H m ) for allm, (45) 

*— » 

which is a discrete convolution. In order that eq. (45) conform to eq. 
(44), the auxiliary sequence {H m } must satisfy 

H m = for m£J, (46) 

and 

y m = for mG J. (47) 

To accomplish this, the values of H m for m e J are determined such 
that these constraints are forced to be satisfied. This is always possible 
since there are Ni + N 2 + 1 free parameters, H m , and the same number 
of conditions on V m . From the above, it follows that 

Uk-Vk for k£J. (48) 

This is easily proved by subtracting eq. (45) for m^ J from eq. (44). 
Now define the sequence {Mk 1 } such that 

X M k M^- k = 8 m , . (49) 

It can then be shown that 

V k = (1 - U ) £ (R m - H m )Mk- m for all k . (50) 

m— — oo 
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Since V* ■ for k E J, we conclude from eq. (50) that 

2 H m M^m= l RmM^m for * G J , 
m— oo m=— oo 

and since if m = for m ^ J, it follows that 

I H m M^ m = S k ,o-NoM^ for keJ, (51) 

me<7 

where we used eq. (43) and eq. (49) for the right-hand side. Eq. (51) 
can be solved if the Toeplitz matrix generated by the sequence {Mk 1 } 
is not singular. This is always the case when M(co) = R(u) + No is 
bounded away from zero and infinity, i.e. for all systems of practical 
interest. 10 

For evaluation of the mse, we need 

I U k R- k = I VkR-k , (52) 

k$j k 

where the equation holds because V* = for k £ J. Since J always 
contains the number zero, we conclude from eq. (43) that R-k can be 
replaced by M-k. This yields together with eq. (52) and eq. (45) 

X VkR-k - (1 - UoHRo - Ho) . (53) 

We use eq. (53) together with eq. (22) to obtain 

U ° = TTTW (54 > 

N + Ho 



which finally leads to 

e " ,,l = a ~ a No~+jLr 

the desired main result. 



2 W> 



(55) 



APPENDIX B 

Analysis of the One-Sided Canceler of Infinite Length 

For iVi finite and N 2 = oo, the set of equations (24) which determines 
Uk reads as follows: 

X UkM m -k - (1 - U )R m for m<-Ni. (56) 

k<-N t 

To solve this one-sided convolution, we factor the sequence {M m } into 
a causal part {Mm} and an anticausal part {M m }\ i.e., 

Mm = l MtMm-n , (57) 

n=0 
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where 

A/: = for n<0 (58a) 

Mn=0 for n>0. (58b) 

We now define a sequence Y„ such that 

£ M + n Y m - n = (1 - U )R m for aU m . (59) 

m— 

Now insert eq. (57) into eq. (56) to obtain 

£ (/* £ M:M^-*-„ = (1 - (/o)i? m for ttK-Nl (60) 

A<-N, n-0 

In addition, substract eq. (60) from eq. (59) for m < -JVi and obtain 
the following set of equations 

Y m = I I/*M m _* for m<-Ni. (61) 

a<-jv, 

Multiply eq. (61) by M™, sum over all m < -N u and use eq. (57) on 
the right-hand side to obtain 

I Y m Mt m = £ UkM-k. (62) 

k<-N, k<-N t 

Recall that 

M k = R k + iV 5*.o , (63) 

and compare eqs. (57) and (59) which determines Y m as 

Y m = (1 - U )Mm for m * . (64) 

Now insert eq. (64) into eq. (62) and make use of eq. (63) once more to 
obtain. the one-sided sum which is required in eq. (53): 

X R- k U k = (l-Uo) £ M~ m M + - m . (65) 

k<-N x m<-Ni 

Since Mm = M- m , it can be shown that 



£ R- k Uk-a-Uo) s 1^ 

k<-N, m<-N, 



- 12 



= (1 - Uo) I \M + m \ 2 . (66) 



m>N, 



Also, insert eq. (66) into eq. (53) and use eq. (63) to find 

N + Ho = Mo- S \M1n\ 2 . (67) 
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With eq. (57) evaluated for m = 0, we finally obtain 



N + Ho= i \MZ\ 2 , (68) 

m-0 



and with eq. (55) we get our desired result, 

€opt = O a ~]yj • (69) 

I \M + m \ 2 

m-0 
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